Deep Learning and Bidirectional Optical Flow Based Viewport Predictions for 360° Video Coding

The rapid development of virtual reality applications continues to urge better compression 360° videos owing the large volume content. These are typically converted 2-D formats using various projection techniques in order benefit from ad-hoc coding tools designed support conventional video compression. Although recently emerged standard, Versatile Video Coding (VVC) introduces specific tools, it fails prioritize user observed regions videos, represented by rectilinear images called viewports. This leads encoding redundant frames, escalating bit rate cost videos. In response this issue, paper proposes a novel framework for VVC which exploits viewport information alleviate pixel redundancy regard, bidirectional optical flow, Gaussian filter and Spherical Convolutional Neural Networks (Spherical CNN) deployed extract perceptual features predict By appropriately fusing predicted viewports on projected Regions Interest (ROI) aware weightmap is developed can be used mask source introduce adaptive changes Lagrange quantization parameters VVC. Comprehensive experiments conducted context Test Model (VTM) 7.0 show that proposed improve bitrate reduction, achieving an average saving 5.85% up 17.15% at same quality measured Viewport Peak Signal-To-Noise Ratio (VPSNR).